Supporting  Observers  During  Distributed  Team  Training  -  The 
Development  Of  A  Mobile  Evaluation  System 


Marcel  P.W.  Van  Berio,  MSc 
Annemarie  M.F.  Hiemstra,  MSc 
Ing.  Wytze  Hoekstra 

TNO  Human  Factors,  Department  of  Training  and  Instruction 

PO  box  23 

3769  ZG  Soesterberg 
The  Netherlands 
E-mail:  vanberlo@tm.tno.nl 


Summary 

Distributed  team  training,  often  in  joint  settings,  is  becoming  more  and  more  important  in  the  military 
training  today.  As  the  teams  and  training  staff  are  not  physically  on  the  same  location,  special  attention 
should  be  paid  to  performance  measurement  and  feedback.  The  question  is  what  should  be  measured  in 
distributed  training  and  how  the  training  staff  should  be  supported  in  doing  this.  This  paper  describes  a 
mobile  performance  measurement  and  evaluation  tool,  specifically  developed  for  distributed  team  training 
(MOPED).  The  MOPED  tool  helps  the  evaluator  in  observing  team  performance  and  in  quickly  generating, 
sending  and  receiving  data  to  support  his  evaluation. 

Introduction 

Distributed  team  training,  often  in  joint  settings,  is  becoming  more  and  more  important  in  military  training 
today.  As  the  subteams  and  team  members  are  not  physically  on  the  same  location,  performance 
measurement  and  providing  feedback  could  be  problematic.  In  order  to  give  adequate  feedback,  it  is 
essential  that  observers,  who  are  distributed  themselves  as  well,  can  quickly  compare  and  integrate  their 
observations.  In  this  way,  the  time  needed  to  prepare  the  after  action  review  can  be  reduced  to  a  minimum. 
The  sooner  the  results  of  an  exercise  can  be  evaluated,  the  better  it  is.  In  a  distributed  context,  there  should 
be  a  relation  between  the  performance  of  the  own  team  and  the  higher  level  team  in  order  to  evaluate  and 
improve  the  co-ordination  and  teamwork.  Although  systems  for  automated  team  performance  measurement 
are  becoming  more  available,  it  is  still  the  human  observer  that  is  responsible  for  measuring  team  processes. 
However,  measuring  team  processes  is  a  difficult  task.  Besides,  the  risk  of  having  several  observers  is 
multiple  interpretations  of  the  same  observed  behavior.  Therefore,  the  issue  is  what  should  be  measured  in 
distributed  training  and  how  the  training  staff  should  be  supported  in  doing  this.  This  paper  describes  a 
mobile  performance  measurement  and  evaluation  tool,  specifically  developed  for  distributed  team  training 
(Hiemstra,  Van  Berio  &  Hoekstra,  2003).  The  tool  helps  the  evaluator  in  observing  team  performance  and  in 
quickly  generating,  sending  and  receiving  data  to  support  the  evaluation.  In  the  next  section,  the 
characteristics  of  distributed  teams  will  be  described,  as  well  as  the  performance  measurement.  Next,  the 
methods  and  results  of  specifying  the  mobile  tool  will  be  illustrated.  This  is  followed  by  a  brief  discussion 
about  the  first  try-out  of  MOPED.  Finally,  some  concluding  remarks  will  be  made. 


1.  Characteristics  of  distributed  teams 

A  distributed  team  consists  of  several  subteams  and/or  individuals  working  together  on  a  joint  mission,  but 
who  are  not  able  to  meet  face-to-face  because  of  differences  in  space  and/or  time.  Distributed  teams  are 
characterized  by  the  fact  that  they  are  geographically  distributed,  electronically  linked,  and  functionally 
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and/or  culturally  divers  (Schraagen,  2001).  In  many  respects,  distributed  teams  are  not  different  from  teams 
conducting  their  tasks  in  a  face-to-face  environment.  In  both  cases,  the  teams  are  serving  the  organization  in 
achieving  a  certain,  common,  goal.  However,  installing  distributed  teams  makes  certain  aspects  of  team 
performance  more  explicitly  visible:  who  is  responsible  for  what,  how  is  the  balance  between  self-direction 
and  central  management,  how  can  teamwork  be  optimized  between  all  subteams  and  individuals.  Preparation 
before  task  performance  as  well  as  a  common  understanding  on  the  various  roles  (convergence)  seem  to  be 
crucial  factors  in  the  success  of  distributed  teams  (Rocco,  1998).  The  same  holds  true  for  the  use  of 
technological  tools  that  links  the  team  members.  The  diversity  of  teams  can  have  a  significant  impact  on  the 
team’s  performance.  Especially  in  the  case  of  joint  missions,  involving  teams  with  different  organizational 
or  national  backgrounds,  cultural  differences  can  cause  many  problems.  Multicultural  teams  should  therefore 
explicitly  formulate  group  norms  and  identify  how  these  cultural  differences  could  affect  the  cohesiveness 
and  effectiveness  of  the  team  (Van  Vliet,  2003). 

Leadership  in  a  distributed  environment  is  critical  in  both  the  planning  and  the  execution  phase  of  the 
mission  (Cook  &  Klumper,  1999).  The  emphasis  should  be  on  relationships  within  the  distributed  team,  or 
with  other  words,  on  communication  and  inteipersonal  skills.  The  leader/commander  should  be  open  for 
advise,  and  he  should  have  the  competency  to  adopt  other  team  members’  perspectives  (Cook  &  Klumper, 
1999).  This  is  difficult  in  a  distributed  environment  because  the  context  is  incomplete.  Team  leaders  should 
therefore  explicitly  indicate  what  issues  they  want  to  be  informed  about,  when  they  want  to  be  involved,  and 
to  what  extent  team  members  can  make  decisions  on  their  own  (Van  Bree,  Vlietman  &  Wierda,  2001). 

A  team  can  be  effective  on  two  levels:  the  task  and  the  process  level.  On  the  task  level,  performance 
assessment  is  relatively  easy  because  the  task  result  can  be  measured  objectively  (e.g.  completion  of  a 
mission,  operating  the  equipment).  This  is  more  difficult  for  processes  within  a  team  or  between  teams. 
Previous  research  has  shown  that  communication,  information  exchange,  initiative/leadership  and  supporting 
behavior  (Smith-Jentsch  et  al.,  1998)  are  important  dimensions  of  team  performance  on  the  process  level. 
Besides,  Orasanu  (1990)  showed  that  better  performing  teams  are  better  planning  their  actions.  These 
aspects  seem  even  more  important  for  distributed  teams,  due  to  the  lack  of  face-to-face  contact.  Singer  et  al 
(2001)  investigated  the  differences  between  local  and  distributed  teams.  They  found  that  in  all  cases  the 
distributed  teams  did  not  outperform  the  local  teams.  Their  explanation  for  this  result  was  the  lack  of 
informal,  and  face-to-face,  communication  between  the  missions.  Finally,  Singer  et  al.  (2001)  stress  the 
relevance  of  communication:  because  of  the  geographical  distribution  of  team  members,  the 
interdependency  increases,  and  therefore  the  need  to  communicate  in  order  to  achieve  the  shared  and 
common  goals. 


2.  Team  performance  measurement  and  feedback 

To  improve  the  training  effectiveness,  performance  measurement  and  feedback  should  be  conducted 
systematically  and  in  a  standardized  manner  across  all  distributed  teams  (Dwyer  et  al.,  1997;  Fowlkes  et  al., 
1994).  As  the  teams  and  training  staff  are  not  physically  on  the  same  location,  special  attention  should  be 
paid  to  performance  measurement  and  feedback.  Observations  of  the  processes  within  the  distributed  team 
should  be  consistent  among  the  various  (distributed)  observers  in  order  to  provide  consistent  feedback  to  the 
team  members.  The  question  is  what  should  be  measured  in  distributed  training  and  how  the  training  staff 
should  be  supported  in  doing  this?  Previous  research  has  shown  that  communication,  information  exchange, 
team  leadership  and  supporting  behavior  (Smith-Jentsch  et  al.,  1998)  are  important  dimensions  of  team 
performance.  These  dimensions  are  even  more  important  for  distributed  teams,  due  to  the  lack  of  face-to- 
face  contact.  In  addition,  preparation  before  task  performance  as  well  as  a  common  understanding  of  the 
various  roles  (convergence)  seem  to  be  crucial  factors  in  the  success  of  distributed  teams  (Rocco,  1998). 

Previous  research  on  team  performance  measurement  resulted  in  the  Command  and  Control  Process 
Measurement  Tool  (C2PMT;  Van  Berio  &  Schraagen,  2000;  Schraagen  &  Van  Berio,  2001).  The  C2PMT  is 
a  generic  checklist  comprising  standards  a  command  &  control  team  should  meet.  Every  standard  is  briefly 
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clarified  and  explained  in  order  to  ensure  a  uniform  interpretation  by  the  evaluators:  it  describes  the  contents 
and  coverage  of  the  standard  and,  if  applicable,  the  relation  with  other  standards.  For  every  standard, 
performance  indicators  have  been  formulated  giving  concrete  form  to  the  standard  enabling  the  evaluators  to 
observe  and  inteipret  the  team  processes.  These  performance  indicators  are  formulated  concisely,  and  are 
easily  scored  in  terms  of  whether  the  behavior  was  observed  or  not.  The  evaluator  can  explain  and  illustrate 
every  observation:  this  contains  both  positive  and  negative  examples  being  observed.  Inclusion  of  these 
example  behaviors  is  important  for  providing  feedback  in  the  final  written  report  and  for  enhancing  learning 
opportunities.  The  C2PMT  was  adjusted  for  distributed  team  training  (C2PMT-Distributed).  With  this  tool 
the  evaluator  can  score  targeted  behaviors  that  are  both  important  within  the  team  and  between  the 
distributed  teams.  Based  on  previous  research  on  the  Mobile  Aid  for  Training  and  Evaluation  (Pruitt  et  al., 
1997;  Lyons  &  Allen,  2000),  the  C2PMT-D  was  implemented  on  a  hand-held  device.  The  functional 
specifications  of  this  MOPED  tool  (MObile  Performance  measurement  and  Evaluation  of  Distributed  team 
training)  will  be  discussed  in  the  next  section. 


3.  Functional  specifications  of  the  MOPED  tool 

The  MOPED  tool  comprises  several  generic  categories  on  which  the  performance  of  a  distributed  team  can 
be  assessed.  Every  category  is  given  concrete  form  by  specific  performance  indicators.  Completeness  and 
usability  of  the  performance  indicators  are  balanced  as  well  as  possible.  If  possible,  redundancies  are 
avoided  in  order  to  prevent  the  same  behavior  being  rated  in  several  categories.  In  order  to  offer  a  tool  that 
can  easily  be  managed  by  observers,  the  total  number  of  categories  has  been  kept  to  a  minimum.  The 
categories  are  described  below,  illustrated  with  some  performance  indicators. 

a)  Preparation:  availability  and  quality  of  plans  and  checklists,  concurrence  of  the  plans  and  checklists 
between  all  distributed  teams. 

b)  Information  flow  within  a  team:  modality  and  timing  of  information  exchange. 

c)  Information  flow  between  teams:  modality  and  timing  of  information  exchange,  informing  the  right 
persons/teams. 

d)  Use  of  ICT  systems:  adequately  using  the  communication  devices,  attunement  of  various  ICT  systems. 

e)  Team  decision  making:  indicating  expectations  by  overall  commander,  stimulating  participation  by 
asking  suggestions,  clear  and  unambiguous  decisions,  stating  the  right  priorities. 

f)  Active  monitoring  of  critical  tasks:  check  deadlines,  asking  for  clarifications. 

g)  Interpersonal  relations  and  mutual  support:  every  team  is  actively  involved,  good  atmosphere  within  the 
team,  supporting  each  other  without  neglecting  one’s  own  task. 

h)  Back  up  facilities:  appointment  of  alternate  commander,  no  interruption  of  the  chain  of  command,  back 
up  of  teams. 

The  MOPED-tool  should  not  only  facilitate  the  performance  measurement,  but  the  debriefing  as  well.  In  this 
respect  it  is  important  to  know  how  the  quality  of  the  own  team’s  performance  relates  to  the  performance  of 
the  other  teams  within  the  distributed  context.  However,  especially  during  large-scale  exercises  with  teams 
physically  dispersed  over  a  large  area,  the  communication  and  information  exchange  between  the  also 
distributed  observers  is  difficult.  Therefore,  the  MOPED  tool  should  facilitate  the  mutual  data  exchange 
between  observers,  and  be  able  to  receive  and  present  these  data  in  a  format  to  be  used  during  the  debrief. 
Based  on  these  demands,  a  pocket-PC  has  been  chosen  rather  than  a  PDA  or  tablet-PC.  The  display  of  a 
PDA  was  considered  to  be  too  small,  while  a  tablet-PC  was  regarded  as  too  large  to  handle  during  field 
exercises.  A  pocket-PC  can  be  easily  carried,  the  display  is  large  enough  to  get  a  clear  overview,  and  a 
keyboard  facilitates  making  notes/explanations.  Figure  1  shows  a  picture  of  the  MOPED  tool.  In  the 
remainder  of  this  section,  the  functionality  of  the  MOPED  tool  is  described.  A  more  detailed  discussion  can 
be  found  elsewhere  (Hiemstra,  Van  Berio  &  Hoekstra,  2003). 
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Figure  1:  The  MOPED  tool 

The  MOPED  tool  consists  of  six  parts,  displayed  as  buttons  on  the  opening  screen  (see  Figure  2).  Selecting  a 
button  by  tapping  the  stylus  on  the  touch  screen,  activates  the  respective  part  of  the  program: 

New:  the  observer  logs  in,  and  indicates  that  a  new  session  will  start. 

Questionnaire:  the  checklist  with  performance  indicators. 

Grading:  overview  of  gradings. 

AAR:  overview  of  observations  indicated  as  relevant  for  the  after  action  review. 

Send  data:  transmitting  data  to  a  central  database. 

Results:  observers’  results  of  other  teams,  received  ant  integrated  by  central  database. 


Figure  2:  Opening  screen  of  the  MOPED  tool 


Figure  3  shows  a  screen  dump  of  the  C2PMT-D  questionnaire.  On  the  left-hand  side,  the  buttons  with 
numbers  1-8  direct  the  observer  to  the  eight  categories  the  C2PMT-D  is  comprised  of.  The  category  that  is 
selected  is  depicted  on  the  display.  Every  performance  indicator  can  be  checked  as  either  observed  (Yes)  or 
not  observed  (No).  In  case  the  observer  wants  to  include  a  performance  indicator  in  the  AAR,  he 
checkmarks  the  AAR-button.  On  the  right-hand  side,  the  observer  explains  and  illustrates  the  rating:  this 
contains  both  positive  and  negative  examples  being  observed.  Tapping  on  an  explanation-box  presents  a 
pop-up  window  enabling  the  observer  to  type  in  more  elaborate  text.  Inclusion  of  these  example  behaviors  is 
important  for  providing  feedback  in  the  final  written  report  and  for  enhancing  learning  opportunities. 
Completely  to  the  right  are  two  arrows  enabling  the  observer  to  scroll  up  and  down  through  all  performance 
indicators.  After  having  filled  out  all  performance  indicators,  an  overall  grading  can  be  determined  for  the 
respective  category  (EX:  excellent,  ST:  satisfactory,  MA:  marginal,  UN:  unsatisfactory,  NG:  not  graded). 
During  the  course  of  an  exercise,  the  observer  can  easily  switch  between  all  categories  by  tapping  on  the 
eight  numbers.  Tapping  on  the  Ready-button  leads  the  observer  back  to  the  opening  screen. 
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Figure  3:  The  C2PMT-D  questionnaire 

All  checkmarks  and  explanations  made  by  the  observer  are  automatically  linked  to  the  other  parts  of 
MOPED.  The  overall  gradings  are  linked  to  the  ‘Grading’  overview.  This  overview  depicts  every  category 
and  its  overall  grading,  providing  for  a  concise  summary  of  the  team’s  performance  (see  Figure  4). 


Figure  4:  The  Grading  overview 

After  an  exercise,  the  observer  can  send  his  observational  data  to  the  central  database  (‘Send  data’).  In  this 
central  database,  all  observational  data  of  all  distributed  observers  are  gathered  and  integrated.  Only  results 
that  are  explicitly  related  to  the  performance  of  the  distributed  team  and  not  a  local  team  (inter  vs.  intra)  are 
sent  back  to  the  observers  in  the  field.  Figure  5  shows  a  screen  dump  of  the  ‘Results’.  The  numbers  behind 
every  performance  indicator  show  how  many  observers  scored  ‘Yes’  or  ‘No’.  The  box  behind  it  show  the 
remarks  and  explanations  made  by  every  other  observer.  Tapping  on  it  will  enlarge  this  box. 
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Figure  5:  The  Results  overview 


In  case  the  observer  wants  to  include  a  performance  indicator  in  the  after  action  review,  he  checkmarks  the 
AAR-button  behind  it.  An  overview  of  all  these  performance  indicators  is  provided  in  the  AAR-part  of 
MOPED.  This  overview  helps  the  observer  in  preparing  and  conducting  the  after  action  review.  Figure  6 
shows  a  screen  dump  of  the  AAR-overview. 


If)  Concurrence  of  plans  and  checklists  between  all 
distributed  teams 


2b)  Brief  and  relevant  communication 


2e)  Information  exchange  without  unnecessary  delay 


3a)  Requesting  crucial  information  if  not  provided  in  time 


"rfes 


'rfes 


No 


No 


Ready 


- El 

Prevented  misunderstanding 


Report  too  late 


Figure  6:  The  AAR  overview 


5.  Try-out 

A  try-out  has  been  conducted  at  the  Operational  School  of  the  Royal  Netherlands  Navy  for  a  period  of  two 
weeks  (for  a  more  detailed  discussion,  see  Van  Rijk,  Hiemstra  &  Hoekstra,  2003).  The  team  was  the 
command  central  team  responsible  for  the  defense  of  a  frigate.  This  team  is  divided  into  two  teams  who  have 
to  work  together,  but  have  their  own  tasks  as  well:  one  covering  the  Anti  Air  Warfare  domain,  and  the  other 
the  domain  of  Anti  Surface/Subsurface  Warfare.  Although  the  teams  share  the  same  command  central  on 
board,  during  off  shore  training  exercises  they  use  two  separate  simulators.  Simultaneously  training  the 
distributed  teams  using  a  shared  virtual  environment  and  the  same  training  scenario  provided  the  opportunity 
to  conduct  a  small  scale  try-out  of  the  MOPED-tool  as  well  as  the  C2PMT-D  questionnaire.  For  this 
purpose,  the  generic  version  of  the  C2PMT-D  has  been  adapted  to  this  specific  domain  and  this  specific 
Navy  command  team.  Further,  a  less  dichotomous  way  of  scoring  has  been  implemented:  in  the  try-out 
version,  performance  indicators  can  be  scored  on  various  scales  rather  than  only  ‘yes’  or  ‘no’.  Besides  on  the 
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pocket-PC,  MOPED  was  also  be  installed  on  a  tablet-PC.  In  this  way,  the  advantages  and  disadvantages  of 
the  man  machine  interface  and  the  usability  of  the  hand  held  computers  could  be  evaluated.  Four  trainers 
tested  MOPED  during  2  weeks  in  32  training  sessions.  Data  were  collected  with  a  questionnaire, 
observations  and  interviews. 

It  was  the  first  try-out  of  MOPED,  so  it  was  inevitable  to  have  several  disadvantages.  The  most  important 
one  was  that  the  participants  suggested  improving  the  debrief  facility.  MOPED  offers  a  fixed  sequence  of 
items  based  on  the  ranking  in  the  questionnaire.  The  participants,  however,  wanted  to  sequence  and  cluster 
the  items  themselves.  With  respect  to  the  Pocket  PC,  it  was  difficult  to  keep  an  overview.  A  disadvantage  of 
the  Tablet  PC  was  the  weight  and  the  heating  up  after  a  while. 

One  of  the  advantages  was  that  MOPED  enables  a  quick  data  exchange  between  the  observers.  Further,  the 
team’s  result  was  immediately  computed,  and  MOPED  supported  a  quick  arrangement  of  items  for  the 
debriefing.  Further,  the  tool  was  easy  to  use,  with  no  flipping  around  of  paper  pages.  With  respect  to  the 
C2PMT-D  items,  the  results  were  not  clearly  interpretable:  most  items  were  perceived  as  valuable,  but 
during  several  training  sessions  the  performance  measurement  appeared  to  focus  only  on  the  commanders  of 
the  two  teams,  meaning  that  not  all  items  of  our  questionnaire  were  actually  filled  out. 


6.  Conclusion 

The  C2PMT-D  is  a  generic  method  that  can  improve  the  quality  of  the  observations  made  during  distributed 
mission  training  because  of  the  standardized  format  in  which  targeted  behaviors  are  scored.  Implementation 
of  this  method  in  the  MOPED  tool  supports  the  observers  in  quickly  generating  and  processing  performance 
data  and  organizing  the  after  action  review.  Based  on  the  experiences  with  the  C2PMT  (Van  Berio  & 
Schraagen,  2000;  Van  den  Bosch  &  Van  Berio,  2002),  applying  the  MOPED  tool  will  have  the  following 
advantages: 

•  A  clear  insight  into  the  command  and  control  process  of  distributed  teams. 

•  An  objective  assessment  of  the  distributed  team’s  command  and  control  process  based  on  the 
performance  indicators. 

•  Support  of  relatively  inexperienced  observers. 

•  The  results  of  an  exercise  are  easier  to  interpret. 

•  The  results  of  various  exercises  are  mutually  comparable. 

•  The  lessons  learned  can  be  determined  more  easily. 

•  Follow-on  actions  can  be  determined  in  a  more  structured  way. 

The  MOPED  tool  provides  the  means  for  gathering  facts  and  data  concerning  the  quality  of  command  and 
control  processes,  and  therefore  has  the  potential  to  determine  follow-on  actions  not  exclusively  related  to 
training  issues,  but  also  to  the  real-life  (quality)  management  of  organizational  processes.  Possible  follow-on 
actions  could  relate  to,  for  instance,  the  quality  and  availability  of  checklists,  the  attunement  of  various  ICT 
systems,  and  the  security  awareness  of  the  personnel. 


7.  Further  Research 

Further  research  will  be  conducted  on  the  development  and  validation  of  the  various  parts  of  the  MOPED 
tool.  At  the  moment,  the  MOPED  tool  is  a  prototype  with  a  generic  version  of  the  C2PMT-D.  Try-outs 
during  several  exercises,  combined  with  experiments,  are  required  to  improve  the  quality  and  usability  of 
both  the  method  and  the  tool.  This  will  bring  us  to  a  better  understanding  of  performance  measurement  and 
feedback  in  distributed  team  training,  and  a  validation  of  the  method. 

Further,  we  want  to  improve  the  support  for  the  AAR.  In  the  current  version  of  the  MOPED  tool,  the 
organization  of  the  after  action  review  is  supported  by  the  ‘Grading’  and  the  ‘Results’  overviews,  showing 
the  results  of  the  team’s  performance  in  isolation  and  in  the  context  of  the  distributed  team.  This  covers  the 


RTO-MP-HFM-101 :  Paper  14 


Page  7  of  1 0 


content  of  the  AAR,  but  not  the  process  of  the  debriefing  itself.  Further  research  will  be  conducted  on 
methods  to  facilitate  and  guide  an  after  action  review  of  a  distributed  team,  and  how  these  methods  could  be 
supported  by  the  MOPED  tool. 

Finally,  integrating  the  MOPED  tool  into  existing  technological  frameworks  is  another  possibility.  One  such 
framework  is  Mind,  developed  by  the  Swedish  Defence  Research  Agency  (FOI).  The  Mind  visualization 
framework  has  been  developed  based  on  the  need  to  handle  large  amounts  of  data  pertaining  to  diverse 
aspects  of  distributed  rescue  operations  (Morin,  2002).  Mind  uses  data  collected  in  the  real  environment, 
regardless  if  it  is  a  live  operation  or  a  training  exercise,  to  construct  a  time-synchronized,  discrete-event 
representation  of  the  course  of  events  of  the  operation.  The  resulting  model  is  a  multimedia  representation  of 
the  distributed  tactical  operation  that  can  be  presented  in  a  visualization  tool  that  supports  time-based 
navigation  and  animation  using  multiple  views.  Mind  has  been  used  for  supporting  after-action  reviews  in 
different  settings  (Crissey,  Morin  &  Jenvald,  2001;  Thorstensson  et  al.,  2001).  TNO  Fluman  Factors  and  FOI 
are  currently  in  the  process  of  combining  their  expertise  of  human  factors  knowledge,  computer  science  and 
training  experience.  Further  research  could  aim  at  the  integration  of  the  Mind  system  and  the  MOPED  tool  in 
order  to  enhance  the  quality  and  speed  of  the  data  collection  and  analysis  (Van  Berio  et  al.,  2003).  This 
should  result  in  achieving  the  optimal  combination  of  automated  performance  measurement  and  feedback 
and  human  observations. 
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